618 research outputs found
Characterizing and Subsetting Big Data Workloads
Big data benchmark suites must include a diversity of data and workloads to
be useful in fairly evaluating big data systems and architectures. However,
using truly comprehensive benchmarks poses great challenges for the
architecture community. First, we need to thoroughly understand the behaviors
of a variety of workloads. Second, our usual simulation-based research methods
become prohibitively expensive for big data. As big data is an emerging field,
more and more software stacks are being proposed to facilitate the development
of big data applications, which aggravates hese challenges. In this paper, we
first use Principle Component Analysis (PCA) to identify the most important
characteristics from 45 metrics to characterize big data workloads from
BigDataBench, a comprehensive big data benchmark suite. Second, we apply a
clustering technique to the principle components obtained from the PCA to
investigate the similarity among big data workloads, and we verify the
importance of including different software stacks for big data benchmarking.
Third, we select seven representative big data workloads by removing redundant
ones and release the BigDataBench simulation version, which is publicly
available from http://prof.ict.ac.cn/BigDataBench/simulatorversion/.Comment: 11 pages, 6 figures, 2014 IEEE International Symposium on Workload
Characterizatio
Quality at the Tail
Practical applications employing deep learning must guarantee inference
quality. However, we found that the inference quality of state-of-the-art and
state-of-the-practice in practical applications has a long tail distribution.
In the real world, many tasks have strict requirements for the quality of deep
learning inference, such as safety-critical and mission-critical tasks. The
fluctuation of inference quality seriously affects its practical applications,
and the quality at the tail may lead to severe consequences. State-of-the-art
and state-of-the-practice with outstanding inference quality designed and
trained under loose constraints still have poor inference quality under
constraints with practical application significance. On the one hand, the
neural network models must be deployed on complex systems with limited
resources. On the other hand, safety-critical and mission-critical tasks need
to meet more metric constraints while ensuring high inference quality.
We coin a new term, ``tail quality,'' to characterize this essential
requirement and challenge. We also propose a new metric,
``X-Critical-Quality,'' to measure the inference quality under certain
constraints. This article reveals factors contributing to the failure of using
state-of-the-art and state-of-the-practice algorithms and systems in real
scenarios. Therefore, we call for establishing innovative methodologies and
tools to tackle this enormous challenge.Comment: 9 pages, 4 figure
- …